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This report by the General Accounting Office (GAO) 
generated by congressional concern over the accomplishments of 
educational research, reviews the activities and impact of the 
contractors (educational laboratories and research and 
development — R6D centers) supported by the Cooperative Research Act 
under the direction of the U.S. Office of Education and, 
subsequently, the National Institute of Education (NIE) • The 
laboratories were to translate the research results of the R&D 
centers into products such as books, audiovisual materials, 
procedures, and organizational structures. After testing and refining 
the products, they were to make them available to local school 
systems, generally through commercial publishers. GAO found that a) 
there was little evidence that products had significant impact on 
classrooms; b) cqmpleted or substantially developed products 
generated little publisher interest; and c) management problems, such 
as frequent changes in Office of Education leadership, have hindered 
the program. The report recommends that NIE a) require the contractor 
to state clearly the objective of the product in terms of 
anticipated, s'pecific educational changes; b) establish evaluation 
requirements for contractors; c) set standards for follow-up 
evaluations; d) monitor contractors* evaluation processes; and e) 
demonstrate the marketability of a proposed product. (BHD) 
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To the Speaker of the House of Representatives 
and the President pro tempore of the Senate 

This report concerns our review of educational 
research and development performed by regional educational 
laboratories and research and development centers. Activi- 
ties performed by these institutions are authorized by the 
Cooperative Research Act, as amended (20 U,S,C, 331), and 
are administered by the National Institute of Education, De- 
partment of Health, Education, and Welfare. 

We made our review pursuant to the Budget and Accounting 
Act, 1921 (31 U.SoC. 53), and the Accounting and Auditing 
Act of 1950 (31 U.SoC. 67). 

We are sending copies of this report to the Director, 
Office of Management and Budget, and to the Secretary of 
Health, ^ Education, an4 Welfare. 




Comptroller General 
of the United States 
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DIGEST 



WHY THE REVIEW WAS MADE 

Because of congressional concern 
over educational research and its ac- 
complishments, GAO reviewed the ac- 
tivities of five educational labora- 
tories and three development centers, 
supported by the Department of 
Health, Education, and Welfare (HEW), 
which had expended about $82 million 
since their inception. 

Basic facts 

In 1963, the Office of Education 
(OE), under the Cooperative Research 
Act, as amended, began supporting 
university-based organizations 
(centers) committed to researching 
significant educational problems. 
The research was to contribute to- 
ward the understanding and improve- 
ment of educational practices. 

In 1965, the Congress authorized OE 
to support the establishments i"" 
dependent, nonprofit institutions 
(laboratories) designed to make the 
results of innovation and experimen- 
tation in education readily avail- 
able to schools^ 

The laboratories were to develop the 
research results into products— such 
as books, audiovisual materials, pro- 
cedures, and organize* i"ional struc- 
tures—that could be used in class- 
rooms; test and refine these prod- 
ucts; and make them available to 



local school systems, generally 
through commercial publishing com- 
panies. 

During GAO's review, the Congress 
enacted the Education Amendments 
of 1972, which established the 
National Institute of Education 
within HEW. In August 1972, this 
organization, became responsible for 
most of the educational research, 
including the laboratory and center 
programs, formerly administered by 
OE. 

Since 1963 Federal appropriations for 
the laboratory and center programs 
totaled about $211 million. As of 
December 1972, 11 laboratories and 9 
centers were engaged in educational 
research and development. 

FINDINGS AND CONCLUSIONS 

Laboratories and centers (referred to 
as contractors) have put forth much 
effort to improve American education 
and have established a pool of personnel 
specializing in research and develop- 
ment of educational products. 

To have an impact, educational re- 
search and development programs 
should result in products which have 
certain essential characteristics. 
They should be able to achieve de- 
sired objectives and be readily and 
economically made available to the 
classroom. 
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Although the contractors have devel- 
oped some products— particularly 
those dealing with teacher training— 
which have been disseminated to the 
intended users, the products GAO re- 
viewed generally did not possess 
these characteristics* (See p. 9.) 

Product evaluation 

OE delegated to the laboratories and 
centers the responsibility for evalu- 
ating the products but did not pre- 
pcjre guidelines setting forth re- 
quirements for sound evaluation. 

Although contractors reviewed by GAO- 
made some form of evaluation before 
disseminating the products, tie 
evaluation processes varied signif- 
icantly among the contractors c^nd 
generally were not adequate to en- 
able them to determine the effec- 
tiveness of their products. (See 
p. 11.) 

Contractor^ generally had not 

—stated product objectives in meas- 
urable terms (see p. 12)> 

—established adequate controls over 
factors affecting the validity of 
their test results (see p. 13), 

—compared their results with the 
results of simlar products being 
used in schools (see p. 14), 

—designed their evaluations to de- 
termine product impact on student 
learning (see p. 15), and 

—provided potential users with 
timely and informative evalua- 
tion reports for use in making pur- 
chasing decisions (see p. 15). 

There was little evidence that con- 
tractor products have had a signif- 
icant impact in classrooms. Followup 



evaluations had not been planned or 
made to determine the long-term im- 
pact of products on the educational 
user. 

OE considered followup evaluations 
desirable but did not require them 
because it emphasized the need to 
cease funding a product once it had 
been disseminated so that funds could 
be reprogramed to new priorities. 
{See pp. 17 to 19.) 

OE consultants employed to independ- 
ently and objectively evaluate con- 
tractor products have generally 
criticized the products as not having 
been proved effective. (See p. 19.) 

A well-defined stotement of a pro- 
posed product's objectives in terms 
of specific educational changes ex- 
pected from product use is essential 
to making an evaluation. A sound 
evaluation system should include 

—a plan for evaluating during de- 
velopnent the extent to which the 
product meets its objectives and 

—a plan for followup studies 3 when 
appropriate and feasible, to deter- 
mine whether the product performed 
inthe classroom as intended. 
(See p. 11.) 

Such a system would enable the Na- 
tional Institute of Education and 
its contractors to have a basis for 
making more informed decisions on 
future program direction and invest- 
ments of the limited Federal funds 
available for educational research 
and development. 

Product mcocketabilitij 

OE intended that contractors' prod- 
ucts be disseminated to the educa- 
tional community by organizations 
other than the contractors— generally 
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commercial publishers. It did not 
require its contractors to assess 
market needs and constraints and to 
contact publishers before product 
development to determine a product *s 
marketability. (See p» 23.) 

6A0 reviewed the major products; in 
17 programs costing $48.8 miliion 
ar»d found that most products which 
were developed or substantially com- 
pleted had generated little publisher 
interest. (See p. 24.) 6A0 ana- 
lyzed contractor product files and 
found that the following factors oc- 
curred most frequently with respect 
to publisher disinterest. 

— -Prodijcts were based on existing 
copyrighted material for which m 
release had been obtained* (See 
P- 24.::) 

—Products were not in a form readily 
usable without contractor assist- 
i ance. (See p. 26.) 

—Products were aimed at a special- 
ized or limited audience. (See 
p. 28.) 

If OE had required assessments of 
market needs and early contacts with 
publishers to help determine consumer 
need or interest, the above problems 
might have been revealed early 
enough to modify or redirect the con- 
tractors* efforts. 

j Management py*oblems 

/ OE provided substantial funds to lab- 
oratories and centers to develop im- 
proved educational products. In the 
beginning stages, OE emphasized prod- 
uct development almost to the exclu- 
sion of product marketing and evalua- 
tion. 

As the contractors matured, they rec- 
ognized the need to focus on these 



other processes, and some contrac- 
tors began identifying procedures 
necessary to carry them out. This 
growth, however, has been gradual 
and has suffered from inadequate 
guidance from OE. 

Certain unanticipated problems, such 
as frequent changes in OE leadership, 
also have affected the successful 
operation of the programs. (See 
p. 9.) 



RECOMMMDATIONS AND SUGGESTIONS 



The Secretary of HEW should direct 
the National Institute of Education 
to: 

—Require contractors to state objec- 
tives in terms of specific educa- 
tional changes expected from using 
the products. (See p, 21.) 

—Establish basic requirements for 
contractors to use in their evalua- 
tions, including, as a minimum, the 
requirement to (1) evaluate, upon 
completion of product development, 
the extent to which the product 
achieved its objectives, (2) main- 
tain control over factors which 
could. affect the validity of evalua- 
tion, results, and (3) make compara- 
tive evaluations when practicable. 
(See p. 21 .) 

—Establish standards for followup 
evaluations, when appropriate and 
feasible, to determine a product* s 
long-term impact in the classroom. 
(See p. 21 •) 

—Monitor the contractors' evaluation 
processes. (See p. 21 .) 

--Demonstrate the proposed product's 
jnarketability, considering such 
factors as the special needs of the 
intended users, the product 
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competition, and the product cost. 
(See p. 32.) 

"Develop alternative ways of dis- 
seminating the products., (See 
p. 32.) 



AGEIJCY ACTIONS AND UNRESOLVED ISSUES 

HEW concurred in GAO's recommenda- 
tions and described actions taken or 
planned to implement them. (See 
pp. 21, 22, and 32.) 



MATTERS FOR CONSIDERATION i 
BY THE CONGRESS 

This report provides the Congre^is 
with information on the laboratories' 
and centers' progress in developing 
and disseminating educational prod- 
ucts and the additional steps the Na- 
tional Institute of Education needs 
to take to improve the programs. GAO 
believes that this report will be use- 
ful to the congressional committees 
having oversight responsibilities for 
this new organization. 




CHAPTER 1 



INTRODUCTION 

The Office of Education (OE) , Department of Health, 
Education, and Welfare (HEW), has stated that progress in 
education depe-nds on research, development ^ demonstration, 
evaluation, and dissemination of new educational products 
and practices • OE has funded a wide range of activities 
designed to seek solutions to educational problems and to 
develop programs to meet students' needs. This support 
covered short-term projects, comprehensive development under- 
takings, and commitments to insure a firm research manpower 
and institution base. Included in the support were the educa 
tional laboratory and research and development center pro- 
grams established under the Cooperative Research Act, as 
amended (20 U.S. C. 331). . 

We reviewed five educational laboratories' and three 
research and development centers' activities to determine 
whether the programs were achieving their objectives and, 
if not, what improvements v;ere needed. These programs gen-' 
erally were designed to improve educational practices in orie 
or more of the following areas-^ teacher- training techniques, 
instructional materials and methods, curriculum development, 
and school organization and management structures. 

During our review, the President signed Public Law 
92-318, Education Amendments of 1972, which established the 
National Institute of Education (NIE) within HEW (20 U.S.C. 
1221e (Supp. II, 1972)). In August 1972, NIE assumed re- 
sponsibility for most of the educational research activities, 
including the laboratory and research and development center 
programs, formerly administered by OE. 

Under the law, NIE is charged with 

--helping to solve or alleviate problems in American 
education ; 

--promoting the reform and renewal of American education 

--advancing education a.s an art, science, and profes- 
sion; 



--strengthening the scientific and technological 
foundations of e ducat ion ;-^ahd 

--building an effective educational research and devel- 
opment system* 

We believe that the information in this report will 
help NIE carry out its responsibilities, 

RESEARCH AND DEVELOPMENT CENTERS 

Two research and development centers were established 
in fiscal year 1964 to conduct coordinated indepth research 
relating to major educational problems. Other centers were 
later established, and, as of December 1972, nine centers 
were receiving NIE support to perform basic research to help 
improve and better understand educational practices. These 
centers, based at institutions of higher education, receive 
funds from these institutions to supplement the Federal sup- 
port. 

EDUCATIONAL LABORATORIES 

The Cooperative Research Act, as amended by the Elemen- 
tary' and Secondary Education Act of 1965 (78 Stat. 44), au- 
thorized the Commissioner of Education to support research 
and development of nonprofit institutions and organizations, 
including educational laboratories. The House and Senate 
committee reports on this legislation indicated that these 
national and regional facilities would be designed to make 
the results of innovation and experimentation in education 
readily available to the schools. 

Laboratories are multidisciplinary and multifunctional 
and conduct activities ranging from research and development 
of new educational products and practices to demonstration 
and dissemination of the results. The laboratories were to 
work with local schools. State departments of education, 
universities, and other groups to translate the research 
results into forms that could be effectively used in class- 
rooms and made available to local schools. 

The development activities of the educational labora- 
tories were designed to complement rather than duplicate the 
centers' research and initial development activities • 
Centers and laboratories frequently work on different stages 



o£ efforts to resolve major educational problems, dividing 
their resp.onsibilities according to staff competencies to 
reach mutual goals, i 

In May 1967, the National Advisory Committee on Educa- 
tional Laboratories stated that the laboratories should con- 
centrate on development and emphasize measures . specif ically, 
designed to convert research knowledge as rapidly as pos- 
sible into educational practice.. In addition, the committee 
recommended that OE support orderly expansion of the labo- 
ratories ' work. 

OE organized a network of 20 nonprofit institutions 
during 1965 and 1966 to operate as regional educational 
laboratories. Federal funding was discontinued in 1969 for 
five laboratories and for four more in 1971 because (pf bud- 
get limitations and OE ' s dissatisfaction with" their per- 
formance. These nine laboratories had received about 
$24 million. As of ^December 1972, the Ather 11 laboratories 
were receiving NIE research and development support. 

FUNDING 

The following table shows that the Congress appropriated 
funds totaling $211,2 million from the time the educational 
laboratory and research and de\'elopment center programs were 
begun through fiscal year 1972. 



Appropriated funds 

Fiscal year Laboratories Centers 

(millions) 



1954 


$ " 


$ 1.0 


1965 




2 ♦ 2 


1966 


8,7 




1967 


17.7 


a 


1968 


,22.9 


12.4 


.1969 


23.4 


9.8 


1970 


25.1 


8.9 


1971 


23.9 


7.2 


1972 


22.6 


9.5 


Total 


$144^ 


$66.9 



^Amount covers 12 to 19 months. Contracts for teven centers were extended 
from 1 to 7 months in addition to the basic 12-month period to have all 
centers on the same contract year. Appropriations were increased .to accoin 
modate this transition.' ' i 
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The five laboratories and three centers included in 
our review had spent about $82 million for research, evalua- 
tion, and dissemination as of November 1972, OE negotiated 
contracts annually with laboratories and centers on the 
basis of their budget requests and plans detailing the scope 
of their work. In recent years OE established priorities 
for the type of educational research and development to be 
carried out, but it still approved funding on the basis of 
requests from laboratories and centers. 

NIE plans to shift from supporting the laboratories and 
centers as institutions to supporting research for developing 
specif ict educational programs. NIE plans also to issue re- 
quests for the work to be done, and all segments of the 
educational community- - laboratories , centers , colleges , 
universities, and individuals--will be eligible to submit 
proposals for NIE funds. 
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CH APTER 2 

T HE IMPACT OF EDUCATIONAL 

RESEARCH AND DEVELOPMENT 

The Federal Government has provided substantial support 
to educational laboratories and research and development 
centers, and these institutions have put forth ijiuch effort 
to improve American education. The laboratories and centers, 
generally referred to as contractors, have successfully es- 
tablished personnel specializing in research and development 
of educational products. They have also developed some pro- 
ducts- -particularly those dealing with teacher training- -which 
have been disseminated to the intended users. 

The contractors generally have had problems, however, 
in getting research results into the classroom. The con- 
tractors, OE, or other interested parties often were unable 
to determine whether the products provided effective educa- 
tional alternatives because tlie contractors had not adequately 
evaluated the products. Also, contractors have not been 
able to interest publishers in marketing some of their edu- 
cational products because they did not adequately consider 
factors affecting marketability, such as product complexity 
and cost. 

One of the causes of the contractors' problems is that 
the climate in which they operated was not always conducive 
to providing effective products. In the early stages of the 
programs, OE emphasized product development almost to the 
exclusion of product marketing and evaluation. As the con- 
tractors matured, they recognized the need to focus on these 
other processes, and some contractors began identifying 
procedures necessary to carry them out. This growth, however, 
has been gradual and, for the most part, without adequate 
OE guidance. 

OE officials believe that the programs have also suf- 
fered from frequent changes in OE management personnel re- 
sponsible for administering educational research and develop- 
ment. Four different management officials were responsible 
for directing OE's research and development program from 
1968 to 1971. In addition, OE officials have stated that 
the organization responsible for monitoring the programs 
within OE was not adequately staffed to carry out its 
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responsibilities. According to OE officials, operating 
fumds were occasionally curtailed to the point that even the 
external reviews of the program which were required by legis- 
lation were cut below the level considered prudent for sound 
management. 

NIE was established to help solve the problems asso- 
ciated with educational research and development. We believe 
that, if federally funded education research and development 
activities are to have an impact, NIE must insure that the 
products demonstrate that they can (1) achieve desired ob- 
jectives and (2) be made readily and economically available 
to the classroom. 
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CHAPTER -3 



OPPORTUNITIES TO IMPROVE PRODUCT EVALUATION 

The overall goal o£ OE*s research programs is to improve 
American education by providing educators with a v;ide array 
of effective alternatives to existing educational programs. 
According to OE, educational research and development 
requires a series of carefully programed efforts to produce 
tested and effective materials, procedures, and organiza- 
tional forms needed to improve specific elements of instruc- 
tion and the educational process. Although the contractors 
included in our review made some form of product evaluation, 
they generally did not convincingly and objectively demon- 
strate their products* effectiveness in terms of the benefits 
to be derived by potential users, especially students. 

The contractors generally had not stated product objec- 
tives in measurable terms. The evaluation proce.sses varied 
among the contractors and were generally not adequate to 
enable them to evaluate their products' effectiveness. The 
evaluation reports which were issued for OE's and users' 
benefits were generally not timely or complete, and contrac- 
tors did not perform followup studies to determine the long- 
term effect of products that had been disseminated. Several 
OE consultants who had been asked to review contractor activ- 
ities also criticized the contractors' evaluations. 

ELEMENTS OF A SOUND EVALUATION 

OE delegated the responsibility for evaluating products 
to the laboratories and centers, but it did not develop 
guidelines setting forth fundamental requirements for a 
sound evaluation process. Contractors were not required to 
and did not state their proposed products' objectives in 
terms of specific educational changes expected from product 
use. A well-defined statement of objectives is essential to 
establishing an evaluation system which should include 

--a plan for evaluating the extent to which the product 
meets the stated objectives and how the product com- 
pares to existing alternatives available to the educa- 
tor and 
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--a plan for followup studies, when appropriate and 
feasible, to determine whether the product performed 
as intended after it was disseminated, and used in the 
classroom. 

Such a system would provide NIE and its contractors a "basis 
for making more informed decisions on future program direc- 
tion and investments of the limited Federal funds available 
for educational research and development.^ 

UNCLEAR PRODUCT OBJECTIVES 

Contractors reviewed generally had not established prod- 
uct objectives concerning the type and degree of changes that 
could be expected from using their products. Because product 
expectations were vague and were not stated in measurable 
terms, we believe it would be difficult, if not impossible, 
to determine product ef f ectiveness • Two examples are dis- 
cussed below. 

1. One contractor was developing an instructional pro- 
gram in individualized learning which another contractor 
originally started. This program was to develop products 
which would provide individualized instruction in reading, 
spelling, mathematics , and science. About $8.4 million had 
been expended on this program through November 19 72. 

The objectives for the products in all of the learning 
areas were stated in imprecise and immeasurable terms. For 
instance, the contractor's objective for its mathematics 
products was stated as follows: 

"Outcome: Improved math program through 
increased prescription power." 

2. Another contractor was developing a mathematics 
product for secondary level students. The objective for this 
product was: 

ti* * * to develop a mathematics curriculum for 
students; of grades 7-12 which is sound and 
appropriate, based on fundamental unifying math- 
ematical concepts and individualized through a 
proper mix on independent study and teacher 
taught materials ." 
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In jboth o£ these examples, the contractors could have 
stated the products' objectives in terms o£ the part icip^mts ' 
expected rate o£ achievement. These students' actual 
acliievement could then be measured 'agains t these objectives 
to see if the products were achieving the intended results . 

SHORTCOMINGS IN EVALUATING PRODUCTS 

In testimony before the Senate Committee on Appropria- 
tions for fiscal year 1973 describing the iinportance of eval- 
uation, the Secretary of MEW stated that evaluation is one of 
HEW's major tools to indicate how to improve its programs. 
The Secretary pointed out that evaluation is intended to pro- 
vide information on what does and does not work so that HEIV 
can get the- maximum impact from its resources. 

Validity of test results 

Many of the contractors in our review did not adequately 
control factors which could affect the validity of test 
results--such as the number and type of test participants, 
test settings, measurement variables, and application of test 
materials--when they performed the product evaluations. It 
is generally recognized among evaluation experts that these 
factors must be controlled to obtain useful evaluation 
results. Two example.^ of the absence of adequate control 
over evaluations are discussed below. 

1. One contractor conducted numerous evaluation studies 
on its reading and mathematics products. These products were 
geared to individualized instruction of students in grades 
.one through six. The tests were conducted at 6 test and dem- 
• onstration schools which at the end of school year 1971-72 
had received over $2.5 million from the contractor to cover 
their costs for acting as demonstration schools. For each 
test school, another school in the same community was 
selected as a control group. Students in the control groups 
did not use the contractor's reading and mathemtics prod- 
ucts . 

The contractor, however, did not adequately control its 
evaluation. For example, the progress of individual stu- 
dents was not reported even though the products were 
directed to meeting the needs of individual students, and 
the accumulative achievement gains of the test groups and 
control groups were not compared. 
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In its progress reports, the contractor stated that test 
students achieved as well as or better than control students 
on standardized tests. We were advised, however, that when 
the contractor administered standardized tests, it allowed 
both the control groups and test students to take the entire 
test rather than only those portions applicable to their 
grade levels as required by the test manual. And all stu- 
dents were allowed to take as much time as they wanted in 
completing the test, which was also contrary to the test man- 
ual. Under these circumstances, results from the tests would 
not be comparable to standardized test norms. 

2. Another contractor developed a 3-year reading pro- 
gram for Alaskan native children. The contractor planned a 
3-year field test of the program beginning in the fall of 
1968 using test groups and control groups composed of stu- 
deiits selected from several schools that were ultimately to 
be'-^given the tested program. The contractor planned to dis- 
seminate the final tested products to all users in the fall 
of 19 72, after allowing 1 year to incorporate revisions 
expected to result from evaluating the test program. In 
19 70, however, the contractor yielded to pressures from State 
school officials and made the program available to substan- 
tially all the native children. The contractor thus lost the 
possibility of comparing test students against control stu- 
dents who had not been exposed to the program. About 
$1.1 million was expended on this program as of November 
1972, 

Comparison studies 

Contractors generally did not perform studies comparing 
the results of their products against (1) the results of sim- 
ilar products already being used in the schools or (2) the 
results of other contractors^ products. OE recognized in 
196 5 that means had to be devised to compare new educational 
products Avith existing products but OE did not require the 
contractors to attempt such comparisons. This information 
would be useful to the educational user in selecting prod- 
ucts from among available alternatives and to NIE in making 
decisions on future product development efforts. 

An example of the absence of comparison studies involved 
a contractor which developed a reading product at a cost of 
about $700,000 to provide supplementary material to a commer- 
cially available reader. Use of the commercial reader alone 

* ERIC ,4 



required the user to spend $4 a student annually on consum- 
able materials. The contractor's product required an addi- 
tional $5 a student annually. The contractor did not cojnpare 
the results obtained using its product (commercial product 
plus supplemental material) to the results obtained using the 
coiamercial reader alone; therefore, potential users of the 
contractor's product did not have data for making an informed 
decision about the product's cost effectiveness. 

Design of studies 

In cases where products were developed to improve cer- 
tain skills of teachers, the evaluations ivere to answer such 
questions as whether the teachers learned the skills and 
whether the teachers believed the training was valuable. The 
evaluations were not concerned with measuring the impact on 
student learning as a result of the teacher's using the prod- 
uct. 

OE's consultants stated the following about one teacher 
training program: 

"Evaluation seems to stop with determining 
teachers' mastery of skills, but there is no 
assessment * * * of change in the typical class- 
room behavior of teachers . There is no emphasis 
on subsequent changes in pupils after the 
teacher training. This omission is unacceptable 
in a program which bases its importance on the 
education of pupils," 

PROBLEMS IN PRODUCT EVALUATION REPORTS 

One of the most important elements in the evaluation is 
the evaluation report. To be used in decisionmaking, the 
reports should be timely and complete. In a number of 
instances, contractors had disseminated their products even 
though their evaluation reports did not meet any of these 
criteria. 

Timeliness 

Evaluation data can only be helpful in making effective 
purchasing decisions if it is provided to the potential user 
in advance of those decisions. Two contractors disseminated 
. products without providing a formal evaluation report. In 
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some cases a report was provided many months after the prod- 
uct was released for commercial marketing, and in other cases 
no reports were provided. For example, a contractor had dis- 
seminated four products to train teachers in certain skills. 
The evaluation reports for three of the products were not 
available as of August 19 72 even though the products had been 
on the market 18 to 32 months. The evaluation report for the 
remaining product was issued 12 months later. Only limited 
quantities of these products were sold. It seems likely that 
more timely availability of evaluation reports demonstrating 
the effectiveness of the products would increase sales. The 
contractor informed us that in the future it would not dis- 
seminate products before it provided an evaluation report. 

Completeness 

The test population for a basic electricity course 
developed and evaluated by one contractor was to comprise 
130 students. However, complete evaluation data was avail- 
able for only 22 of the 82 ' tudents who completed the course 
and this information was the basis for the evaluation report. 
The contractor did not state why complete data was not avail-' 
able for all 130 students. Such information'may have signif- 
icantly affected evaluation results. 

The same contractor issued an evaluation report on its 
3-year reading program (see p. 14) indicating that the test 
students had met one of the objectives- -to read a commercial 
third grade reader- -after completing the program. However, 
the contractor did not report that the tests were adminis- 
tered to only 45 of the. 101 students who completed the pro- 
gram. Although a total of 767 students were involved in the 
pilot and field testing, the contractor's evaluation report 
did not contain information on students who were not tested, 
those who were detained in a grade, or those who were dropped 
from the program. Information concerning the reasons stu- 
dents did not achieve in a program is as essential for evalu- 
ating the effectiveness of the program as information on stu- 
dents who did achieve. 

In another contractor's evaluation study on its inner- 
city teaching program, the program graduates were compared 
with graduate^* from a comparison group of teachers who did 
not participate in the program. The contractor reported that 
more of its program graduates were teaching in the inner city 
than graduates from the comparison group. We analyzed data 
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relating to the 19 69-70 program graduates and comparison 
graduates and found that 91, or 61 percent, of the 149 pro- 
gram graduates had stated a preference for teaching in the 
inner city before taking the program, although only 30 of 
the 100 comparison group graduates had stated this prefer- 
ence. However, the contractor did not report this data in 
its study. 

LIMITED PRODUCT IMPACT 

Contractors generally could not demonstrate that the 
products they developed were effective. In some cases the 
contractors' evaluations of teacher- training products showed 
that the teachers had not successfully mastered the skills 
involved. 

For example, a contractor developed, at a cost of 
$800,000, a preservice teacher- training program that was 
intended to better prepare potential teachers for work in the 
inner city. The contractor established and published per- 
formance standards that the prospective teacher should meet 
upon completing the program. The major performance standards 
measured the prospective teacher ' s (1) attitude toward teach- 
ing as a vocation, (2) compatability with a culturally 
deprived school setting, (3) reactions to varied teachijig 
situations, and (4) interaction with pupils. 

To measure a teacher' s attitude toward teaching as a 
vocation, a performance standard of 60 was established for 
participants completing the program. According to the stand- - 
ardized test norms, achieving a raw score of 60 would have 
placed a program participant in the lower one-third of the 
Nation's graduating education seniors. During 3 years of 
field testing, only 50 percent of the program participants 
met this performance standard. 

The majority of the participants met the program's 
standard for measuring reactions to varied teaching situa- 
tions. K'Cwever, no data was provided in the contractor's 
evaluation report showing how many students had achieved the 
stated objectives before starting the program. Our analysis 
of the teacher reaction data showed that the averages for 
the groups tested were generally higher before taking the 
program than after completing the program. 
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The performance standards for measuring a teacher^ s 
interaction with pupils were broken down into seven catego- 
ries. In one category 81 percent of the participants met the 
performance standard, and in two categories 6 2 percent met .| 
the standards. Only 43, 40, 32, and 24 percent, respectively, 
met the four remaining categories' standards. 

On the basis of these evaluation results, it appears 
that the program participants shov^fed a low level of achieve- 
ment from the teacher- training program. 

Standardized tests were generally used to measure stu- 
dent achievement in programs aimed at improving learning. 
However, the contractors were unable to demonstrate that the 
gains made by students using their products were signifi- 
cantly higher than the gains made by students in control 
groups . 

FOLLOWUP EVALUATIONS NOT PERFORMED 

Experts who evaluate education programs generally agree 
that longitudinal studies--sometimes referred to as followup 
studies-'Should be made before definitive statements can be 
made concerning the long-term effectiveness of such programs. 
They point out that the relationship of short-term and long- 
term program effects is often unknown and that data should 
be collected on groups who have received program services and 
on comparison groups who have not received such services over 
a long enough period for possible program effect to appear or 
disappear, or until such data provides reliable indicators of 
long'- term program effects. They recognize that different 
types of programs will require different followup periods to 
provide data needed for making informed decisions concerning 
a program's effectiveness. 

Contractors generally had not planned or made such- lon- 
gitudinal or followup evaluations. Although such evaluations 
may be difficult to conduct for some programs and products, 
they apparently are needed for determining the effectiveness 
of programs and products expected to have a lasting impact on 
school children. 



For example, one contractor was developing an instruc- 
tioi^al program for students who do not succeed in existing 
systems. The program plan indicated that it would overcome 
the "washout" effect of other early childhood educational 
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programs and thus achieve long-term results. The washout 
effect occurs when students respond successfully to the pro- 
grams early but lose the advantages gained after they are 
returned to their original environment for 3 or 4 years. 
Although recognizing the desirability of long-term impact 
studies, a contractor official in charge of evaluations 
stated that the contractor would not perform them on its pro- 
gram because it lacked the necessary resources. In the 
absence of such studies^ the contractor may be unable to 
convincingly demonstrate that the washout effect has been 
effectively eliminated by the use of its program. 

^ OE stated that - followup evaluations were desirable but 
not encouraged because it emphasized the need to cease fund- 
ing a product once it had been disseminated so that funds 
could be reprogramed to new priorities. NIE stated that fol- 
lowup evaluations would be necessary to determine the long- 
term effect of educational products but that such evaluations 
were costly and^could not be supported under its present, 
funding level. 

CONSULTANT EVALUATIONS 

Over the- last several years, OE has contracted with con- 
sultants to independently and objectively evaluate products 
resulting from its support of contractors' activities. The 
three studies that we reviewed genera]^ly criticized the labo- 
ratory and center products, primarily because the products 
had not been properly evaluated. 

One consultant study was performed during 19 70 and 19 71 
to identify products which OE could consider for further dis- 
semination assistance. The contractors submitted 31 products 
for this study which they considered field tested and ready 
for dissemination. The consultants who performed the study 
recommended nine products for dissemination assistance, eight 
of which were contractor products. However, the consultants 
criticized the evaluation results for most of the 31 prod- 
ucts--either because evaluation data was not provided or 
because the data provided showed that the products had lim- 
ited effectiveness. One contractor had submitted 12 prod- 
ucts, none of which were ready for OE dissemination support, 
according to the consultants. The contractor, however, had 
already disseminated 6 of the 12 products. Contractor offi- 
cials told us that they had not received feedback from OE 
about their products as a result of the consultant's study 
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and OE had not required them to correct the matters noted in 
the consultant study. 

A second consul1*ant study was completed in March 1972 
to develop procedures to identify evidence of the impact of 
specific products so that an overall assessment could be made 
of educational research and development's impact. The con- 
sultant team considered 117 products which had emerged during 
the previous 5 years. Fifty-five of the products were feder- 
ally funded, including 17 which were laboratory and center 
products. One of the consultants' objectives *was to find 
evidence of student cognitive gains-- increases in academic 
knowledge. The contractors believed- that 14 of the 17 prod- 
ucts they submitted had resulted in measurable cognitive 
gains in students. The consultants reviewed the contractors' 
evaluation test results and found evidence of cognitive gains 
by students for only six of these products. 

The purpose of the third consultant study, completed in 
late 19 72, was to make recommendations about future funding 
levels. These consultants were also generally critical of 
the contractors' evaluation efforts and recommended reduc- 
tions in funds for some of the programs. 

CONCLUSIONS 

The contractors' evaluation processes were generally not 
adequate to evaluate their products ' effectiveness in terms 
of the benefits to potential users. The contractors had not 

--stated product objectives in measurable terms, 

--established adequate controls over factors affecting 
the validity of test results,, 

--compared the results of their products with the 
results of similar products , 

--provided the educational user with timely and informa- 
tive evaluation reports for use in making purchasing 
decisions , and 

--planned or performed studies of the long-range impact 
of their products on school children. 
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Although OE had given the contractors responsibility for 
conducting the evaluations, OE had not developed guidelines 
setting forth the fundamental elements of a sound evaluation 
process. Properly designed evaluations are necessary if NIE 
and its contractors are to have a sound basis for making 
decisions on program direction and the allocation of limited 
Federal funds for educational research and development. 

RECOMME'i^DATIONS TO THE SECRETARY 
OF HEALTH, EDUCATION, AND WELFAR E 

The Secretary of HEW should require NIE to: 

--Require contractors to state objectives in terms of 
specific educational changes expected from using the 
products . 

'--Establish basic requirements for contractors to use in 
their evaluation processes, including, as a minimum, 
the requirement to (1) evaluate, upon completion of 
product development, the extent to which the objec- 
tives were achieved; (2) control factors which could 
affect the validity of evaluation test results; and 
(3) make comparative evaluations whenever practicable. 

--Establish standards for followup evaluations, when 
appropriate and feasible, to determine a product's 
long-term impact on the classroom. 

--Monitor the implementation of the evaluation processes 
selected by the contractors. 



HEW concurred in our recommendations and stated that NIE 
would require contractors to (1) state objectives, strate- 
gies, and expected outcomes for their products and (2) submit 
a plan establishing milestones and criteria for evaluation, 
including an evaluation upon product completion of the extent 
to which the objectives were met. HEW stated also that NIE 
will monitor the compliance with these evaluation processes 
to insure their adequacy and technical quality and will exam- 
ine additional ways of insuring more effective control over 
evaluation efforts. 
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HEW also stated that NIE is studying the role of 
comparative evaluations and the most effective way in which 
they can be carried out. Further, HEW stated that the exist- 
ing state of the art of evaluation technology is not yet 
sufficiently developed but that NIE will consider establish- 
ing standards for followup evaluations in anticipation of a 
developing evaluation technology. 
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CHAPTER 4 



N EED FOR EFFECTIVE MARKETING GUIDELINES 

AND PRODUCT DISSEMINATION STRATEGIES 

OE intended to have the contractors ' products 
disseminated to the educational community by organizations 
other than the contractors--generally by commercial pub- 
lishers--but it had not required its contractors to determine 
the potential marketability of proposed products. Important 
factors--including assessments o£ market needs and constraints 
and early contacts v;ith publishers- -which the contractors 
should have considered before beginning product development 
were generally not considered until after the product was 
developed- At that point contractors were often not able 
to interest commercial publishers in their products because 
of copyright problems, product complexity, size of potential 
market, and cost factors. As a result, products have been 
delayed in or deterred from getting into the classroom. 

STRATEGY FOR DISSEMINATING PRODUCTS 

OE had delegated to contractors the responsibility for 
arranging for their own product dissemiiiation< Contractors 
had to develop the necessary support systems; identify and 
resolve the barriers that might hinder dissemination; and 
locate publishers, distributors, and teacher trainers. 

In 1965, when the educational laboratory program was 
started, OE established a public domain policy for research 
materials developed with OE funds. This policy provided 
that anyone could publish such materials. OE found, however, 
that its products were not being commercially disseminated, 
one reason being that commercial publishers were reluctant 
to pick up public domain products because they did not want 
to invest time and money in products for which they did not 
have exclusive distribution rights. To accelerate effective 
dissemination of educational materials developed with Fed- 
eral support, OE established a copyright program in 1968 to 
provide copyright protection for products. However, because 
of other problems experienced by contractors, this change 
in policy did not have a significant impact on publisher 
interest. 
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PROBLEMS IK CONTRACTOR MARKETING OPERATIONS 



We reviewed the major products resulting from 17 
research and development efforts involving expenditures of 
S48,8 million in Cooperative Research Act funds as of Novem 
ber 1972. Most products which were developed or substan- 
tially completed had generated little publisher interest. 
IVe analyzed contractor files and found that the following 
factors occurred most frequently with respect to publisher 
disinterest. 

--Products were based on existing copyrighted material 
for which no release had been obtained. 

--Products were not in a form readily usable without 
contractor as sis tance . 

--Products were air^ied at a specialized or limited 
audience . 

If OE had required assessments of market needs and early 
contacts v^ith publishers to help determine consumer need 
or interest, the above problems might have been revealed 
soon enough to permit contractors either to overcome the 
problems or to modify or redirect their efforts. 

Products based on existing copyrighted materials 

Materials which are copyrighted normally carry a state 
ment reserving all rights and prohibiting reproduction with 
out permission in writing from the publisher. If a contrac 
tor uses copyrighted material as part of its own program or 
product, the contractor is required to get a release from 
the copyright owner. In the absence of such a release, a 
contractor can be held liable for damages and be restrained 
from using the copyrighted material. 

If a contractor receives a qualified release- - that is, 
authorization to use the copyrighted material for experi- 
mental purposes only and not for commercial release--it is 
left with three alternatives v/hen a product is ready for 
dissemination. It can (1) obtain the publication services 
of the publisher who copyrighted the original material, 

(2) eliminate the copyrighted material from its product, or 

(3) wait for copyright expiration. 
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The first alternative may not be desirable because the 
publisher may refuse to provide the services or may provide 
the services at a higher cost than the contractor is willing 
to absorb. If copyrighted materials make up a substantial 
part of the contractor's program, eliminating copyrighted 
materials is not an effective alternative, Because copy- 
rights are generally for 28 years with renewal rights, await- 
ing copyright expiration would also not be an effective 
alternative. 

Several products were developed which contained sub- 
stantial copyrighted material • None of these products had 
been published at the completion of our fieldwork. Two 
examples are discussed belotv. 

1, In 1964, a center introduced a commercially avail- 
able individualized reading program into an experimental 
school. This program had been copyrighted by a commercial 
publisher. Over the next several years, the center modified 
the program by adding audiotapes, storybooks, diagnostic 
tests, and a management system to the basic commercial in- 
structional materials. The program was given to a labora- 
tory for field development, testing, and dissemination. The 
laboratory introduced the program into demonstration test 
schools in 1966 and 1967. 

During the initial years of the program, the center 
obtained permission to use the commercial materials for 
experimental purposes. However, neither the center nor the 
laboratory obtained the author's or publisher's permission 
to commercially publish and disseminate the program using 
the copyrighted commercial materials. 

Center officials informed us that, as early as 1967, 
it became evident that the commercial publisher who copy- 
righted the original material would not agree to publish 
the program as changed by the center. Because the center 
believed that its program could not be marketed commercially, 
it then began to develop a new primary reading program. 

While the new program was being developed, the labora- 
tory continued the initial reading program in field-test and 
demonstration schools through the 1971-72 school year. The 
laboratory plans to phase out the initial reading program 
in the demonstration schools and introduce the new primary 

ER?C ,5 



reading program as it becomes available for field testing 
in 1974. As of November 1972, about $710,000 had been ex- 
pended on the initial reading program. 

2. Another contractor was developing an inquiry skills 
program that was intended to improve the skills of teachers 
and students in asking and responding to classroom questions. 
Through November 1972, about $1.7 million of Federal funds 
had been spent on the program. The major component of the 
program is a biology curriculum-oriented product. The con- 
tractor has developed three versions of the product using 
three different commercially available texts. Only the three 
publishers of these texts will be able to use the product in 
its existing form because they hold the copyrights. One of 
the publisliers refused to publish the product because of its 
forecast of a low market potential, and, as of December 1972, 
commitments had not been received from the other two pub- 
lishers . 

Products not in a form readily 
usable without contractor assistance 

For new educational products to have an impact in the 
classroom, they should be in a readily usaL:le form; that is, 
they should be exportable from the contractor to the user. 
But two contractors developed products which could be used 
effectively only if the contractor provided special training 
to the user. At the time of our fieldwork, these products 
had not been widely marketed because the contractors did not 
have the necessary resources for training. These cases are 
discussed below. 

1. Since 1966 a contractor has been developing a pro- 
gram that is intended to provide individualized instruction 
to maximize a student^s ability to understand and use mathe- 
matics. As of November 1972, the development costs were 
estimated at $4.6 million. The program consists of two 
components-- one for all students in kindergarten through 
6th grade, and the other for students ranked in the upper 
20 percent of their classes in the 7th through 12th grades. 

The contractor provided the training needed to teach 
the program material to teachers who tested the product dur- 
ing development. A contractor official advised us that the 
contractor did not have the resources to provide the teacher 
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training on a scale that 
commercially marketed. 



would allow the product to be 



The contractor's advisory committee discussed the prob- 
lem o£ teacher training in 1970 and again in 1971, The com- 
mittee stated that the contractor could not rely on univer- 
sities to provide the needed teacher training for the pro- 
gram. However, a contractor official advised us that the 
contractor chose to continue program development in the hope 
of ultimately devising a way to provide teacher training. 

When we completed our fieldwork, the contractor had not 
been able to devise a means of getting the teacher- training 
component to the potential product users and publishers had 
told the contractor that they could not market the product 
without this component. 

2. A contractor developed a teacher- training program at 
a cost of about $800,000 which was intended to provide poten- 
tial inner-city teachers with a cultural orientation to im- 
prove their skills in dealing with educational problems com- 
mon to the inner- city classroom and a low- income environment. 
The contractor's program was designed to operate for one 16- 
week semester at inner-city elementary and high schools and 
consisted of 8 weeks of orientation, seminars, and visits to 
inner-city homes and schools and 8 weeks of student teach- 
ing. 

The program was begun in 1966 and was first field tested 
in the fall of 1967. During school year 1968-69, the field 
testing was expanded and in July 1971, the contractor made 
available for distribution the completed product of the pro- 
gram, a manual for program operation. We were advised that 
the manual was not a textbook for use by students training 
to become teachers, but instead was for developing a teach- 
ing staff who would instruct prospective inner-city teachers. 

The contractor's executive director informed us that 
the educational change to be brought about by the program 
necessitated that the potential user be given special instruc- 
tion and guidance. But the manual did not contain the needed 
instruction or guidance and it did not contain a disclaimer 
statement to the effect that the program could not be imple- 
mented without the needed training or contractor guidance. 
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The executive director told us that, because funding 
had remained the same for the past 2 years but costs had in- 
creased due to inflation, the contractor decided not to use 
contract funds for distributing its products or for providing 
assistance to potential users of such products. The deci- 
sion was based on an assumption that the products would have 
a reasonable chance to be marketed. 

The contractor obtained a $250,000 Federal grant in 
May 1972 to provide training and assistance in setting up 
8 demonstration sites for showing interested potential users 
how its product worked. The executive director told us that 
the contractor did not have the funds and manpower to provide 
the training and assistance needed for nationwide implementa- 
tion. 

NIE recognizes that contractors generally have not been 
provided sufficient funds to furnish support services to 
schools using their products. NIE plans to consider this 
matter in developing a policy for assisting in the dissemina- 
tion of educational products. 

Products developed for specialized 
or limited audiences 

A number of the contractor products were developed for 
use by special groups, including rural students, Alaska na- 
tives, and Mexican-Americans. The contractors generally 
assumed that publishers would market such products when they 
were ready for dissemination. Publishers were often reluctant 
to market such products, however, because of the relatively 
low publication revenues anticipated. Consequently, products 
have been delayed in getting into the classroom until the 
contractors obtain an alternative distribution channel other 
than publishers. 

For example, by 1970 a contractor had developed, at an 
estimated $373,000, four s elf- instruct ional products for 
rural students to use in studying electricity, welding, 
speech, and plastics. Each of the products is a self- 
contained unit of programed instruction designed so students 
can work at their own pace. 

Contractor officials advised us that, when the self- 
instructional materials were being developed and tested, the 

ERJC 



contractor had not devised a marketing strategy for 
disseminating the products and had not contacted commercial 
publishers. Development o£ the products proceeded on the 
basis o£ the identified need for such products in rural areas. 
The officials had assumed that there would be no problem in 
finding commercial publishers to market the products once 
they were ready for dissemination. 

As of December 1972, contractor efforts to find commer- 
cial firms willing to publish and disseminate the product 
had been unsuccessful. Contractor officials stated that 
publishers were reluctant to pick up the materials because 
their marketing and service efforts were oriented toward 
areas of large population rather than the rural locations 
for which the products were designed. The officials pointed 
out that rural students constituted a small percentage of 
the total student population. 

Products appear too 
costly for the user 

In 1970 the American Association of Publishers reported 
that the annual nationwide expenditure for all educational 
materials averaged $15 a student. We found that products 
costing between $10 and $20 a student annually were developed 
and are being developed by OE contractors for individual 
subject areas. In addition, commercial publishers advised 
us that one of the problems v/ith contractor materials was 
their high cost. Although we did not find an instance in 
which a publisher rejected a contractor product because it 
was too costly, we believe that schools might be precluded 
from purchasing such high cost products without a Federal 
subsidy or without cutting back on existing programs. 

For example, one contractor was developing an individ- 
ualized learning program composed of mathematics, reading, 
and science products. The contractor's estimated per-pupil 
costs for its primary and intermediate individualized reading 
products for the 1972-73 school year are shown in the table 
below. Neither of these products has attracted publisher 
involvements Amounts shown represent first-year, or startup, 
costs but do not include the costs of teacher aides or the 
costs of training administrators , teachers , and teacher aides 
in how to use these products. 
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Subject 



Per-pupil cost--first year 



18 0 -pupil 540 -pupil 900- pupil 
model model model 



Primary reading 

(grades kindergarten 

through three) $79 $56 $52 

Intermediate reading 
(grades four through 

six) 80 63 60 



The contractor estimated that the annual per-pupil cost 
of replacing consumable printed materials for the primary 
reading and intermediate reading products would be approxi- 
mately $9 and $17, respectively. These amounts did not 
allow for replacing nonconsumable items, such as damaged 
tapes or wornout books. In addition, many of the printed 
materials used in the reading products were printed by a 
Federal printing plant and sold at prices established to 
enable the contractor to recover costs incurred for printing, 
collating, binding, and shipping. If such materials were 
commercially published, overhead and profit factors might 
necessitate higher user cost. 

As early as July 1969, consultants hired by OE to make 
onsite reviews were concerned that the high development costs 
of the contractor's individualized learning program could 
impair the products' success. They stated that adopting the 
products would be limited by the financial ability of school 
systems, and they recommended significant reduction in cost. 

The contractor provided $2.5 million to 6 schools to 
test and demonstrate its individualized learning program, 
including its reading products. The support was primarily 
for personnel costs and ranged from $66,000 to $108,000 a 
school in school year 1970-71, the last year the contractor 
provided support to all 6 schools* In school year 1972-73 
the contractor withdrew its support to the demonstration 
schools. Because of the costs involved, four of the six 
schools withdrew from the contractor's program, while the 
remaining two elected to participate in only selected por- 
tions of the program. 

The contractor's executive director agreed with us t.%at 
cost should be considered in product development. He stated, 
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however, that the real question was whether school officials 
and taxpayers were willing to commit their resources to in- 
• sure that children receive the quality education they deserve. 
He also stated that the fiscal difficulties schools Avere 
experiencing had been caused by rising personnel costs 
rather than the rising costs of instructional materials* 

NIE officials agreed that product cost should be a 
major consideration during development. They stated that, 
in some cases, keeping the costs within traditional limits 
might hamper the contractor in its efforts to develop an 
effective educational product. They pointed out that on the 
surface some products may appear to be costly; however, the 
overall costs in installing such products in a school sys- 
tem may be reasonable in terms of the potential long-term 
educational benefits that could result. 

STEPS TAKEN TO DISSEMINATE 
PRODUCTS 

OE established the National Center for Educational 
Communication, which later became part of a NIE task force 
to help disseminate educational products developed under OE 
contracts. Since 1970 OE has provided about $1.46 million 
to help disseminate 6 products developed by laboratories 
and centers. We noted that three of the six products already 
had publishers; therefore, OE has been subsidizing these 
publishers^ marketing costs. NIE officials agreed with our 
observation and told us that this matter would be considered 
in their current study concerning product dissemination. 

CONCLUSIONS 

OE has in the past relied on its contractors to ade- 
quately market their products by obtaining a proper distribu- 
tion channel to get the products to the user. However, 
publishers have generally not been interested in the con- 
tractors^ products and contractors have not devised effec- 
tive alternatives for disseminating their products . 

Before contractors begin to develop a proposed product, 
they should identify alternatives for marketing and dissemi- 
nating their products and should document the rationale so 
that NIE may make informed decisions on whether the strategies 
selected are appropriate for getting the products to their 
potential users. 
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NIE and its contractors must also place greater concern 
on resource investment decisions. Before significant re- 
sources are invested, there should be reasonable assurance 

(1) that proposed products are needed to improve education, 

(2) that they will be developed in forms that can be readily- 
used, and (3) that their costs either will not be out of 
line in comparison to existing products, or if more costly, 
will produce at least commensurate savings in other educa- 
tional expenditures or commensurate benefits to the quality 
of education. 

RECOMMENDATIONS TO THE SECRETARY 
OF HEALTH, EDUCATION, AND WELFARE 

The Secretary of HEW should require NIE to: 

--Demonstrate the marketability of proposed products 
considering such factors as the special needs of the 
intended users, the product competition, and the cost 
of the product. 

--Develop alternative means by which the products may 
be disseminated. 

HEW agreed with our recommendations and stated that, 
under NIE's new policy, it will award contracts to those 
who have most effectively demonstrated their capacity to 
develop products which meet NIE requirements, including such 
elements as the extent to which the proposed products address 
the defined needs, provide attractive alternatives to other 
available products, and can be produced at costs commensurate, 
with their potential value . 

HEW stated also that NIE is conducting a major dissemi- 
nation policy study to develop alternative strategies. 
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CHAPTER 5 



SCOPE OF REVIEW 

Our review was directed toward ascertaining whether the 
benefits anticipated from the laboratory and center programs 
were being achieved and, if not, what improvements were 
needed. We reviewed the legislative history of the labora- 
tory and center programs; OE and NIE program policies and 
directives; and funding applications, reports, and other 
pertinent documents relating to the laboratory and center 
programs. 

We made our review at OE and NIE headquarters in Washing- 
ton, D.C.J at five educational laboratories, and at three 
research and development centers in four States. We in- 
terviewed laboratory and center personnel, commercial pub- 
lishers, teachers, local school officials, and educational 
consultants involved in the laboratory and center programs. 
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DEPARTMENT OF HEALTH. EDUCATION, AND WELFARE 



OFFICE OF THE SECRETARY 



WASHINGTON. D.C. 20201 



AUG 17 1973 



Mr, Morton E. Henig 
Associate Director 
Manpower and Welfare Division 
U.S. General Accounting Office 
Washington, D.C. 20548 

Dear Mr. Henig: 

The Secretary has asked that I respond to your letter of 
May 16, 1973, in which you asked for our comments on a 
draft report to the Congress entitled, "Educational 
Laboratory and Research and Development Center Programs 
Need to be Strengthened," Our comments, are enclosed. 

The opportunity afforded us to comment on this report in 
draft ^form is most appreciated. 



Sincerely yours. 




Charles Miller 

Acting Assistant Secretary, 

Comptroller 



Enclosure 
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I, GAP Recommendation 



HEW should 



Require contractors to state objectives in terms of 
specific educational changes that are expected to result from the 
use of the products. 



Department Coifflnent 



We agree. As the GAO report notes, the Department has shifted 
from supporting the laboratories and centers (i.e., contractors) 
as institutions to a policy of "program purchase," whereby NIE 
support for laboratory and center research and development will 
be provided through individually arranged contracts. Under this 
policy, each laboratory and center contractor is required to state 
objectives, strategies, and expected outcomes for its programs, 
and all curriculum development programs must produce empirical 
evidence of the learning accomplishments of students who use the 
materials in field- test settings. 

II . GAO Recommendation 



HEW should 

— Establish basic requirements to be used by contractors in 
their evaluation processes including., as a minimum, the requirement 
to: 

1. evaluate, upon completion of product development, 
the extent to which the objectives were achieved; 

2. maintain control over factors \rtiich could affect 
the validity of evaluation test results; and 

3. make comparative evaluations whenever practicable. 
Department Comment 



We agree. As the GAO report notes, NIE has begun to develop more 
effective methods of product validation. New laboratory and center 
contracts negotiated in 1972-73 require contractors to evaluate upon 
completion of product development the extent to which the objectives 
have been achieved. 

In addition, each contractor's Resource Allocation and Management Plan 
establishes milestones and criteria for evaluation during the course 
of the project. These plans are reviewed by NIE and adjusted as necessary 
to insure the effectiveness of evaluation activities, and they are used 
by NIE as the basis for monitoring progress and quality of product devel- 
opment . 

Q NIE is also examining additional ways of insuring more effective control 
gp^(^" over evaluation efforts. One study is now underway to determine methods 
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of enabling developers to do more extensive and effective field- 
testing. Another effort is in progress to define more clearly the 
various stages in product development and testing so that more 
meaningful evaluations can be conducted at the various stages. 

We agree with GAO that comparative evaluations between new and existing 
products should be made whenever practicable, although we do not 
necessarily agree that the contractors who develop the products should 
make these evaluations. The role of comparative evaluations and the 
most effective means for their conduct are issues now under study as 
NIE develops its product validation policies and Institute evaluation 
plans. 

III. GAO Recommendation 
HEW should 

Establish standards for follow-up evaluations, where appropriate 
and feasible, to determine a developed product's long-term impact on the 
classroom. 

Department Comment 

We agree that such evaluations are important and encourage their use 
where practicable. As the report points out, however, follcw-up evalua- 
tions are costly; and, even assuming that more than ample funds were 
availab're for their conduct, the state-of-the-art is not yet sufficiently 
developed to provide an adequate return on the commitment of funds. 
Nevertheless, in anticipation of a developing evaluation technology, NIE 
will be considering the establishment of standards for such studies as 
it formulates its product validation policies. 

IV. GAO Recaipmendation 

HEW should 

— Monitor the implementation of the evaluation processes selected 
by contractors. 

Department Comment 

We agree. NIE is now placing special emphasis upon monitoring imple- 
mentation of evaluation processes selected by contractors. Each laboratory 
and center contractor has submitted a Resource Allocation and Management 
Plan containing mi] "ones associated with submission of evaluation re- 
ports and criteria ' « r intermediate and final evaluations. NIE staff are 
now reviewing these e nation designs to insure their adequacy and 
technical quality. As each project progresses, NIE staff will check 
progress against evaluation milestones, review evaluation reports, conduct 
site visits, and require modification of program and evaluation processes 
as appropriate* 
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V, GAP Recommendation 
HEW should 

Demonstrate the marketability of proposed products considering 
such factors as the special needs of the intended users, the product 
competition, and the cost of the product. 

Department Comment 

We ag7:ee. With the Department's shift from institutional support for 
the laboratories and centers to "program purchase", the laboratories 
and centers wilj, apply for awards in competition with other applicants 
from the R&D community at large in response to NIE Requests for Proposals 
NIE will award contracts to those laboratories and centers and other 
applicants which most effectively demonstrate their capacity to develop 
products which meet the NIE requirements, including such elements as the 
extent to which the proposed products address the defined needs, provide 
attractive alternatives to other available products, and can be produced 
at costs commensurate with their potential value. 
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VI- GAP Recommendation 
HEW should 

— Develop alternative means by which products may be 
disseminated* 

Department Comment 

We agree. NIE is now conducting a major dissemination policy 
review consisting of the following: 

(1) policy studies on the feasibility of providing Federal support 
for contractor dissemination, on alteimative copyright and royalty 
arrangements, and on ways to improve incentives for dissemination 
of unusual materials, multi*media systems, and products requiring 
significant changes in local practice; (2) plans to expand knowledge 
of the dissemination process through a case-study analysis of effective 
dissemination strategies and a compilation from empirical data of how 
dissemination actually works; and (3) a conference with the publishing 
industry in which representatives of all groups involved in the develop- 
ment/marketing process will participate. 



ERLC 



APPENDIX II 

PRINCIPAL OFFICIALS OF THE 
DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE 
RESPONSIBLE FOR ADMINISTERING ACTIVITIES 
DISCUSSED IN THIS REPORT 

■ Tenure of office 
From To 



SECRETARY OF HEALTH, EDUCATION, 
AND WELFARE! 

Caspar W. Weinberger 
Elliot L. Richardson 
Robert H. Finch 
Wilbur J. Cohen 
John W. Gardner 
Anthony J. Celebrezze 



COMMISSIONER OF EDUCATION: 
John R. Ottina 
John R. Ottina (acting) 
Sidney P . Mar land, Jr . 
Terrel H. Bell (acting) 
James E. Allen, Jr. 
Peter P. Muirhead (acting) 
Harold Howe II 
Francis Keppel 

DIRECTOR, NATIONAL INSTITUTE 
OF EDUCATION (note a) : 
Thomas K» Glennan 
Emerson J. Elliott 



Feb. 


1973 


Present 


Juno 


1970 


Jan . 


1973 


Jan. 


1969 


June 


1970 


Mar . 


1968 


Jan. 


1969 


Aug. 


1965 


Mar . 


1968 


July 


1962 


Aug. 


1965 


Nov. 


1972 


Present 


Aug. 


1973 


Present 


Nov. 


1972 


Aug. 


1973 


Dec. 


1970 


Nov . 


1972 


June 


1970 


Dec . 


1970 


May 


1969 


June 


1970 


Jan. 


1969 


May 


1969 


Jan. 


1966 


Jan. 


1969 


Dec. 


1962 


^Jan. 


1966 


Nov. 


1972 


Present 


Aug. 


1972 


Nov. 


]972 



ASSISTANT SECRETARY (EDUCATION) : 
Sidney P. Marland, Jr. 



^As of August 1, 1972, the responsibility for administering 
educational research and development activities was tvans- 
ferred from the Office of Education to the National Insti- 
tute of Education. 
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